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Two decades on from the abolition of the binary divide in higher education in Australia, what has happened to the relative research 
performance of institutions that started from quite diverse positions? We use two databases, Thomson Reuters ISI and Scopus, to measure 
growth rates in research output. We find that there has been some convergence in research publications, with the newer universities 
catching up on the traditional research-intensive universities. No Go8 university is in the top quartile of growth rates. In absolute terms, 
however, the output differences remain very large. The findings are not unduly sensitive to the choice of database. 


Introduction 

The abolition of the binary divide in higher education 
in Australia is now two decades old. Institutions have 
now had time to adjust to new mission statements and 
financial incentives. In the 1990s the former colleges 
of advanced education were still adapting to the new 
requirements, especially in research and research train- 
ing. At the same time, many established universities 
were preoccupied with bedding down changes arising 
from amalgamations and takeovers. With the greater 
stabilization of the system in the last decade, now is 
an appropriate time to look at whether there has been 
any convergence in the research performance of Aus- 
tralian universities. 

Research funds have for many years been allocated 
competitively but, more recently, federal govern- 
ments have signalled that performance will play an 


increased role in government funding of research. The 
disbandoned Research Quality Framework (RQF) has 
been replaced by the Excellence in Research for Aus- 
tralia (ERA) initiative. 

With government funding for teaching Australian 
undergraduate students, stagnant and full-fee under- 
graduate places are now not permitted, Australian 
universities can increase revenue from three sources: 
international students, fee-paying postgraduate stu- 
dents, and research funds. An improved research per- 
formance increases income from all three sources: 
directly from research funding allocations, indirectly 
through an improvement in international rankings and 
its effect on fee-paying student demand. 

The financial incentives are for less research-inten- 
sive universities to improve their research output; 
in relative terms this is easier to do from a low base. 
Some convergence in research performance is there- 
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fore expected as, unlike many European and Asian 
countries, federal governments have not chosen to 
fund selected institutions at a higher rate for either 
teaching or research. 

The measurement of research performance is a key 
driver of international rankings and will become of 
greater importance in allocating government research 
funds in Australia with the phasing in of the Sustain- 
able Research Excellence in Australia (SRE) initiative. 
It follows that the attributes of the databases used 
to measure performance are of growing importance. 
Two databases dominate the measurement of research 
performance: the ISI Web of Knowledge, provided by 
Thomson Reuters, and Scopus, provided by Elsevier. 

The two most objective measures of international 
research performance are based on Thomson Reuters' 
ISI: Shanghai Jiao Tong (SJT) and the rankings by the 
Higher Education Evaluation and Accreditation Coun- 
cil of Taiwan (HEEACT). The other popular ranking 
has been Times Higher Education-QS which, although 
based primarily on surveys, in recent years used 
Scopus to derive its quantitative measures. From 2010 
the Times Higher Education rankings are to be totally 
rethought and the database will change to Thomson 
Reuters. QS (Quacquarelli Symonds) will provide sep- 
arate rankings using the Scopus database. Within Aus- 
tralia, Scopus is the database used in the ERA exercise, 
the results of which are planned to feed into the SRE 
initiative. 

Because of the dominant positions of these two 
databases we evaluate them as alternative measures of 
research output of Australian universities. Does one 
database favour a particular type of institution or do 
they yield similar results? 

Data 

We measure research performance by publications in 
the form of articles, conference papers and reviews. 
The three categories are listed separately in ISI Web 
of Science and in Scopus. We choose to work with 
the aggregate figure as there is some arbitrariness in 
allocating publications between the three categories. 
Also, the aggregate publication figure for ISI Web of 
Science can be cross-checked against the data from 
another Thomson Reuters product, ESI. ESI is specifi- 
cally designed to provide information on institutions, 
whereas ISI Web of Science (and Scopus) is designed 
primarily for locating publications on topics and the 
work of individual authors. 


The databases were interrogated by searching the 
affiliations of authors. Care needs to be taken with 
universities that have international namesakes, such as 
Newcastle, Victoria and New England, and those with 
overlapping names, such as the University of South 
Australia and Flinders University of South Australia. It 
is relatively easy to use ISI for our purpose but Scopus 
requires a search over the different styles that can be 
used for affiliation, e g. University of Sydney, Sydney 
Univ. Unfortunately, it was not possible to get reliable 
estimates for Victoria University and this institution 
was deleted from all the analysis. We also exclude non- 
members of Universities Australia, namely, The Uni- 
versity of Notre Dame Australia and the offshoots of 
overseas universities. 

We look at annual output of Australian universities 
over the period 2004-2008, which, allowing for insti- 
tutional and publication lags, will reflect changes in 
government policy over the last decade. The period 
will also begin to pick up the effect on institutional 
policies of international rankings following the first 
SJT rankings in 2003. The importance that institutions 
place on international rankings has been documented 
by Hazelkorn (2007). 

Data are taken from the online data banks as at August 
2009. The Scopus data incorporated the enlarged list 
of journals in the humanities that were added in June 
2009. Scopus included around 15,500 peer-reviewed 
journals, ISI around 12,500 journals. Of course, with 
the growing emphasis in the ERA and elsewhere on 
quality of research, greater coverage per se is not nec- 
essarily an aim in its own right. 

The aggregate output for all Australian universities 
as measured by the two databases is given in Table 1 . 
Overall, Scopus indexed 1 5 per cent more articles pub- 
lished by researchers in Australian universities than 
ISI. Comparing Thomson Reuters data, publications as 
indexed by ISI were a little above those from ESI, but 
the maximum difference, for any institution, over the 
whole period 2004-2008 was 7 per cent. 


Table 1: Australian university publications (‘000) f: 
2004-2008 


Data Base 

Articles 

Conference 

proceedings 

Reviews 

Total 

Scopus 

140.7 

30.3 

18.1 

189.1 

ISI 

122.4 

13.1 

9.1 

144.6 

Scopus/ISI 

1.15 

2.31 

1.99 

1.31 


f Includes all members of Universities Australia except Victoria 
University ( see text). 
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Table 2: Classification of Australian Universities 

Sandstone and Brick (Go8 + Tas) 

Unitechs (ATN + Swinburne) 

Australian National University (ANU)* 

Curtin University of Technology 

Monash University* 

Queensland University of Technology (QUT) 

University of Adelaide* 

RM1T University 

University of Melbourne* 

Swinburne University of Technology 

University of New South Wales (UNSW)* 

University of South Austraba (UniSA) 

University of Queensland* 
University of Sydney* 

University of Technology, Sydney (UTS) 

University of Tasmania* 

New Universities 

University of Western Australia (UWA)* 

Australian Catholic University (ACU) 
Bond University* 

Gumtrees 

Charles Darwin University 

Deakin University 

Charles Sturt University 

Flinders University of South Australia* 

CQUniversity 

Griffith University" 

Edith Cowan University 

James Cook University* 

Southern Cross University 

La Trobe University 

University of Ballarat 

Macquarie University 

University of Canberra 

Murdoch University 

University of Southern Queensland (USQ) 

University of Newcastle* 

University of Western Sydney (UWS) 

University of New England (UNE) 

Victoria University (not included in analysis 

University of Wollongong 

- see text) 

* denotes university had a clinical medical school before 2006. 


Research performance of 
groups of universities 

We first look at the performance 
of groups of universities using the 
two databases. We classify universi- 
ties into four groups, based on the 
nomenclature used by Marginson 
and Considine (2000), except that 
we shift Swinburne University from 
‘New’ to ‘Unitech’. The groups are 
given below and the allocations in 
Table 2. 

• Sandstone or Brick'. Go8 universi- 
ties plus Tasmania 

• Gumtrees: Universities established 
front the early 1960s to mid 1970s 
whose surrounds are typically 
planted with native flora. 

• Unitechs'. Institutions strong in 
technological areas 

• New Universities'. Mainly estab- 
lished after 1987, the post Dawkins era. 

Universities that have a clinical medical program 

have an inbuilt advantage when ranking institutions by 
research performance. For example, over the period 
2004-2008, 23 per cent of publications of Australian 
universities were in clinical medicine, as indexed by 
ESI. We therefore also divide institutions between 
those that have a medical school (with first student 
intakes earlier than 2006) and those that do not. 

Table 3 compares the percentage of Australian univer- 
sity publications that are accounted for by each group 
of institutions over the period 2004-08, using the two 
databases. Using ISI, the Sandstone/Brick universities 
accounted for nearly 68 per cent of all publications; this 
percentage rises to 76 per cent if all institutions with a 
clinical medical school are included. Using Scopus, the 
contribution of the Sandstone/Brick universities was 
three percentage points lower (two per cent lower if 
only articles are included). The obverse of this is that 
the contribution of Unitechs was higher in Scopus than 
in ISI, but in part this is because Scopus included more 
conference proceedings, a form of publication that is 
more common in engineering and related disciplines. 
The Unitechs accounted for 20 per cent of all Australian 
conference proceedings papers in Scopus. 

We now turn to the question of improvements in 
relative performance. In particular, are the newer uni- 
versities catching up? To answer this question we look 


Table 3: Share of Australian publications by university 
groupings, 2004-08 (per cent) 


Group 

ISI 

(total) 

Scopus 

(total) 

ISI 

(articles) 

Scopus 

(articles) 

Sandstone/ 

Brick 

67.9 

64.5 

68.0 

65.7 

Gumtree 

18.4 

18.4 

18.6 

18.8 

Unitech 

8.9 

11.4 

8.5 

9-9 

New 

4.8 

5.7 

4.9 

5.5 


at the share of output of each university group in each 
of the five years 2004-2008. The results are given in 
Table 4 for the two databases. The general finding is 
that the Unitechs and New universities have increased 
their share of total publications at the expense of the 
Sandstone/Brick universities. Under either database, 
the share of publications attributable to the Sandstone/ 
Brick universities has fallen by just under 2 percentage 
points. Thus, there is evidence of some convergence in 
research performance, although the research-intensive 
Sandstone/Brick universities still dominate, producing 
a little under two-thirds of total output in 2008. 

Research performance of individual 
universities 

The use of groupings facilitates analysis by control- 
ling for variables such as age and profile, but within 
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some of the groups there is considerable heterogene- 
ity. In this section we look at performance of individual 
institutions. We again ask: does the choice of database 
matter greatly and which universities have exhibited 
the greater rate of growth in publications? 

The ranking of institutions by absolute output is 
not particularly sensitive to whether ISI or Scopus is 
used as the database. The only noticeable effects are 
that the Unitechs improve their rank when Scopus 
is used (QUT, for example, increases 5 places), and 
Macquarie, Tasmania, James Cook and Flinders each 
fall 4 places when Scopus replaces 1ST The rankings 
at the top (Sandstone/Brick) and the bottom (New) 
are virtually identical using the two databases. At the 
top lie the Go8 universities in the same order: Sydney, 
Melbourne, Queensland, UNSW, Monash, ANU, UWA 
andAdelaide. These rankings are not adjusted for size 
of institution. 

To look at changes in performance we fit exponen- 
tial time trends to the five years of data for each insti- 
tution. Rates-of-growth measures have the advantage 
that they control for profile. The section 3 findings for 
the groupings would suggest that some of the new uni- 
versities are likely to have the fastest rates of growth. 

In Table 5 we present rates of output growth of indi- 
vidual universities grouped into quartiles. We do this 
based on both the ISI and Scopus databases. In gen- 
eral, the two databases produce quite similar results. 
Note, however, that all the rates of growth have some 
upward bias imparted by the inclusion of some Aus- 
tralian journals only in the later years of the period 
covered. As expected, the highest rates of growth are 
exhibited by the New universities and by the Unitechs, 
albeit in several cases the growth is from a very low 
base. No Sandstone/Brick universities appear in the 


Table 4: Annual shares of publications by university 
group, 2004-08 (per cent) 


Group 

2004 

2005 

2006 

2007 

2008 

ISI 






Sandstone/ 

Brick 

68.2 

68.6 

68.3 

68.2 

66.5 

Gumtree 

18.8 

18.2 

18.1 

18.2 

18.8 

Unitech 

8.6 

8.5 

8.8 

8.8 

9.4 

New 

4.3 

4.7 

4.9 

4.8 

5.3 

Scopus 






Sandstone/ 

Brick 

65.6 

64.7 

64.6 

64.1 

63.8 

Gumtree 

18.9 

18.6 

18.4 

18.4 

18.2 

Unitech 

10.3 

11.2 

11.4 

11.7 

12.0 

New 

5.2 

5.5 

5.6 

5.8 

6.0 


top quartile and three are located in the bottom quar- 
tile using either database. 

At least as measured by research output, there is 
evidence of convergence in the research perform- 
ance of Australian universities. The correlation coeffi- 
cients between base output levels in 2004 and rates of 
growth over the period 2004-2008 are negative: -0.30 
for Scopus and -0.29 for ISI, both significant at the 10 
per cent level. 

Publications and other research 
performance measures 

Research performance measures include total publi- 
cations, publications in prestigious journals, citations, 
competitive grants obtained and election of research- 
ers to academies. Publications per se have historically 


Table 5: Annual rates of growth of research publications, Australian universities, 2004-2008. 


Quartile 

ISI 

Scopus 

Ql [highest 
growth] 

ACU, Ballarat, Bond, Charles Darwin, Charles Sturt, 
Curtin, Sunshine Coast, UniSA, Wollongong 

ACU, Ballarat, Bond, Charles Darwin, Charles Sturt, Curtin, 
QUT, Sunshine Coast, UniSA 

Q2 

CQUniversity, Deakin, Griffith, James Cook, 
Melbourne, Monash, QUT, Sydney, USQ 

CQUniversity, Deakin, Griffith, Melbourne, Monash, RMIT, 
UTS, UWS, Wollongong 

Q3 

Adelaide, Edith Cowan, Flinders, Macquarie, 
Queensland, Southern Cross, Tasmania, UTS, UWS 

Canberra, James Cook, Macquarie, Queensland, Swinburne, 
Sydney, Tasmania, UNSW, UWS 

Q4 

ANU, Canberra, La Trobe, Murdoch, Newcastle, RMIT, 
Swinburne, UNE, UNSW, UWA 

ANU, Adelaide, Edith Cowan, Flinders, La Trobe, Murdoch, 
Newcastle, Southern Cross, UNE, UWA 


Notes: 

Universities in same quartile for both data bases are in italics. 

ISI: QI is > 14.5% per year, Q2 is 10.3% to 145%, Q3 is 75% to 10.3% and Q4 is < 75 %. 
Scopus: Ql is >14.0% per year, Q2 is 10% to 14.0%, Q3 is 7.5% to 10%, Q4 is < 75% 
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been important for Australian universities because 
they are used to allocate research funding. 

In previous work (Williams and Van Dyke, 2007, 
2008) we have found high correlations between alter- 
native measures of research performance for Australian 
universities. Australian data, at least at the institutional 
level, tend to confirm a strong empirical regularity 
between citations and publications first observed at 
the international level by Katz (2000). Katz found that 
for both countries and a range of scientific disciplines 
citations increased more than proportionately with 
publications. Large research groups generate more 
citations per paper. More precisely, he found: 

citations = k. publications P 

where (5 > 1. Using data for nations he found (i = 
0.27, but it was a little lower for countries and disci- 
plines. Katz's findings extend to data for Australian 
universities. For example, using ESI data for the period 
2004-2008, the exponent on publications is 1.18 with 
a standard error of 0.04. 

Within academia, the international research standing 
of an institution depends heavily on publications in 
the top journals in the various disciplines. The existing 
databases do lop off the lower-quality tail of journals; 
ISI, for example has well-established criteria for inclu- 
sion in its citation indexes. However, further trunca- 
tion is frequently used for quality measures. 

The importance of adjusting output measures for 
quality depends on the extent to which the distribution 
of output across journals varies between institutions. 
However, there is little empirical evidence on the rela- 
tionship between publications in quality journals and 
total publications. The HEEACT rankings provide data 
on both total ISI publications in the last two years and 
publications in the top 5 per cent of journals within 
each field of study, as measured by ISI journal-impact 
factors. The (Spearman) rank correlation between the 
two series in the 2009 rankings, for the top 100 ranked 
institutions, is relatively high at 0.80. Completion of the 
ERA exercise in Australia will provide further evidence 
on the extent to which measuring quality affects the 
rankings of research performance. 

Concluding remarks 

In this paper we have used two commercially available 
databases, ISI and Scopus, to provide output measures 
of research performance. The results are not partic- 
ularly sensitive to the database used, except that the 


more technologically oriented institutions are favoured 
by the inclusion of more conference proceedings in 
the Scopus database we used. We conjecture that the 
two databases are similarly unlikely to give markedly 
different results for citation counts. 

We found evidence of some convergence across 
institutions in the number of publications produced, 
with the more technological institutions and newer 
universities gaining ground at the expense of the older 
research-intensive universities. But is this an evening 
up or an evening down of research performance 
across institutions? Influenced by their relatively poor 
performance in the international rankings, countries 
such as China, Germany and France have introduced 
differential funding models. Greater concentration of 
research funding on selected research-intensive insti- 
tutions may not increase total research output, but it is 
likely to lead to an improved national presence in the 
international rankings and thus greater recognition of 
the academic standing of a nation’s universities. InAus- 
tralia, rather than differential funding of institutions, 
government policy is directed towards concentration 
of research funding on teams, irrespective of their loca- 
tion. This policy will contribute to an increase in the 
total sum of quality research in Australia but it is an 
open question as to whether such an approach fully 
exploits the synergies that arise from large clusters of 
researchers. 

Ross Williams is a Professorial Fellow in the Melbourne 
Institute, University of Melbourne. 
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